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Abstract Text (1) : 

A high speed remote storage controller system for a computer system has cluster 
nodes of symmetric multiprocessors . A plurality of clusters of symmetric 
multiprocessors each of has a plurality of processors, a shared cache memory, a 
plurality of I/O adapters and a main memory accessible from the cluster. Each 
cluster has an interface for passing data between cluster nodes of the symmetric 
multiprocessor system. Each cluster has a local interface and interface controller. 
The system provides one or more remote storage controllers each having a local 
interface controller and a local-to-remote data bus. A remote resource manager 
manages the interface between clusters of symmetric multiprocessors . The remote 
store controller is responsible for processing data accesses across a plurality of 
clusters and processes data storage operations involving shared memory . A macro^is 
provided for processing a plurality of simultaneous data storage operations either 
synchronously through interaction with a sequential multistage centralized pipeline 
to serialize requests and provide address interlocking services or asynchronously 
whereby main memory accesses bypass a centralized system pipeline. These accesses 
can occur in parallel with other remote storage operations. 



1. A high speed remote storage controller system for a computer system having 
cluster nodes of symmetric multiprocessors , comprising one or more remote storage 
controllers each having an associated local interface controller, and a local-to- 
remote data bus, and the one or more remote storage controllers responsive to a 
remote management system for managing the resources comprising the remote storage 
controller for data accesses between clusters of symmetric multiprocessors which 
are controlled for inter nodal storage operations by one of said remote storage 
controller system, each of which clusters has a plurality of processors, a shared 
cache memory, a plurality of I/O adapters and a main memory accessible from the 
cluster, said remote store controller while processing data accesses across a 
plurality of clusters for data storage operations involving shared memory and inter 
nodal storage operations, enlisting a single one of said remote storage controller 
to perform remote cast outs, store requests from an I/O adapter, main storage 
padding operations, and main memory move page operations and while performing 
remote data storage operations to main memory, said enlisted single remote storage 
controller also handles cross cluster invalidations associated with maintaining 
inter-nodal cache coherency for said computer system having cluster nodes of 
symmetric multiprocessors, and which contains a deadlock avoidance mechanism 
designed to detect inter-nodal deadlocks which normally result from one resource on 
a local cluster waiting for a second resource on a remote cluster which is 
deadlocked against a third resource on said remote cluster waiting for a fourth 
resource on the local cluster which is deadlocked against said first resource on 
the local cluster, and which employs a fast hang quiesce mechanism embedded in each 
remote storage resource which works in conjunction with similar fast hang quiesce 
mechanisms throughout the System Controller to prevent system-wide hangs either 
caused by the failure of the current remote storage resource to make forward 
progress or by the failure of another operation somewhere in the Storage Controller 
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ART-UNIT: 2188 

PRIMARY- EXAMINER: Padmanabhan; Mano 
ASSISTANT-EXAMINER: Ross; John M 
ATTY-AGENT-FIRM: Augspurger; Lynn L. 

ABSTRACT: 

A high speed remote storage controller system for a computer system has cluster 
nodes of symmetric multiprocessors . A plurality of clusters of symmetric 
multiprocessors each of has a plurality of processors, a shared cache memory, a 
plurality of I/O adapters and a main memory accessible from the cluster. Each 
cluster has an interface for passing data between cluster nodes of the symmetric 
multiprocessor system. Each cluster has a local interface and interface controller. 
The system provides one or more remote storage controllers each having a local 
interface controller and a local-to-remote data bus. A remote resource manager 
manages the interface between clusters of symmetric multiprocessors . The remote 
store controller is responsible for processing data accesses across a plurality of 
clusters and processes data storage operations involving shared memory . A macro is 
provided for processing a plurality of simultaneous data storage operations either 
synchronously through interaction with a sequential multistage centralized pipeline 
to serialize requests and provide address interlocking services or asynchronously 
whereby main memory accesses bypass a centralized system pipeline. These accesses 
can occur in parallel with other remote storage operations . 

4 Claims, 10 Drawing figures 
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DOCUMENT- IDENTIFIER: US 6647453 Bl 

TITLE: System and method for providing forward progress and avoiding starvation and 
livelock in a multiprocessor computer system 

Brief Summary Text (14): 

Briefly, the invention relates to a system and method for avoiding "livelock" and 
"starvation" among two or more input /output (I/O) devices competing for the same 
data in a symmetrical multiprocessor (SMP) computer system. The SMP computer system 
includes a plurality of interconnected processors having corresponding caches, one 
or more memories that are shared by the processors, and a plurality of I/O bridges 
to which the I/O devices are coupled. Each I/O bridge includes one or more upstream 
buffers and one or more downstream buffers. An up engine is coupled to the upstream 
buffer and is controls the flow of information, including requests for data, from 
the I/O devices to the processors and shared memory . A down engine is coupled to 
the downstream buffer, and controls the flow of information from the processors and 
shared memory to the I/O devices. A cache coherency protocol is executed in the I/O 
bridge in order to keep the data in the downstream buffer coherent with the 
processor caches and shared memory . As part of the cache coherency protocol, the 
I/O bridge obtains "exclusive" {not shared) ownership of all data fetched from the 
processor caches and the shared memory, and invalidates and releases any data in 
the downstream buffer that is requested by a processor or by some other I/O bridge. 
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ART-UNIT: 2181 
PRIMARY-EXAMINER: Vo; Tim 

ABSTRACT: 

A system and method avoids "livelock" and "starvation" among two or more 
input/output (I/O) devices of a symmetrical multiprocessor (SMP) computer system 
competing for the same data. The SMP computer system includes a plurality of 
interconnected processors, one or more memories that are shared by the processors, 
and a plurality of I/O bridges to which the I/O devices are coupled. A cache 
coherency protocol is executed the I/O bridges, which requires the I/O bridges to 
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obtain "exclusive" (not shared) ownership of all data stored by the bridges. In 
response to a request for data currently stored by an I/O bridge, the bridge first 
copies at least a portion of that data to a non-coherent buffer before invalidating 
the data. The bridge then takes the largest amount of the data saved in its non- 
coherent buffer that its knows to be coherent, and releases only that known 
coherent amount to the I/O device, and then discards all of the saved data. 

11 Claims, 11 Drawing figures 
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DOCUMENT-IDENTIFIER: US 6636926 B2 

TITLE: Shared memory multiprocessor performing cache coherence control and node 
controller therefor 



Brief Summary Text (9) : 

The system according to the second reference described above, in which each node is 
not configured only with a processor having a cache memory, is a multiprocessor 
system in which each node is configured with a processor including a cache memory, 
a memory and an I/O device. This system is what is called a distributed shared 
memory multiprocessor (physically-distributed logicall y-shared memory 
multiprocessor ) , in which the memories and the I/O devices are distributed 
physically among the nodes but shared logically by the nodes. In the system 
according to the second reference, a plurality of nodes are coupled to each other 
by buses for address and coupled by a crossbar switch for data. By use of four 
address buses, four address snoop operations can be performed in parallel. The 
physical address space is divided into four parts so that each address bus can 
snoop different address spaces at the same time. 



1. A shared memory multiprocessor comprising: a plurality of nodes each configured 
with at least one of a processor having a cache memory, a memory device and an I/O 
device, and a node controller; and an inter-node connection network for 
interconnecting a plurality of said nodes; wherein at least one of a plurality of 
said nodes includes said processor and at least one of said memory device and said 
I/O device, and the whole of a plurality of said nodes include at least one said 
processor, at least one said memory device and at least one said I/O device; 
wherein said node controller of each of a plurality of said nodes includes a 
processor unit constituting an interface with said processor in a local node, at 
least a memory unit constituting an interface with said memory device in the local 
node and an I/O unit constituting an interface with said I/O device in the local 
node, a network unit constituting an interface with said inter-node connection 
network, and an intra-node connection circuit for connecting said processor unit 
and at least one of said memory device and said I/O device to said inter-node 
connection network unit; wherein said processor unit includes an inter-unit address 
decode circuit and a cache coherence control circuit, said inter-unit address 
decode circuit decoding the control information and the address information in a 
memory access reguest or an I/O access request issued by the processor in the local 
node, and adding to said access request the information on the node associated with 
the memory unit or the I/O unit intended as a destination of said access request, 
the information on the unit intended as a destination of said access request, and 
the cache coherence control information indicating whether the cache coherence 
control is required or not, said cache coherence control circuit performing the 
cache coherence control of the processor in the local node in the case where the 
cache coherence control information added to the access request received from said 
network unit indicates that the cache coherence control is required; wherein said 
I/O unit includes an inter-unit address decode circuit whereby the node information 
and the unit information for the memory unit or the I/O unit intended as a access 
request destination and the cache coherence control information indicating whether 
the cache coherence control is required or not are added to the memory access 
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request or the I/O access request issued by the I/O device in the local node; 
wherein said network unit includes a transfer unit for transferring the access 
request received from said intra-node connection circuit to said inter-node 
connection network, and a transfer unit for transferring the access request 
transferred thereto from said inter-node connection network to said intra-node 
connection network; wherein said intra-node connection circuit transfers said 
access request to the unit in the local node designated as a destination of 
transfer based on the cache coherence control information, the node information and 
the unit information added to the access request transferred from said inter-node 
connection network through said network unit; and wherein said inter-node 
connection network transfers said access request to the node designated by the 
cache coherence control information and the node information added to the access 
request received from said network unit. 

6. A shared memory multiprocessor comprising: a plurality of nodes each configured 
with at least one of a processor having a cache memory, a memory device and an I/O 
device, and a node controller; and an inter-node connection network for 
interconnecting a plurality of said nodes; wherein at least one of a plurality of 
said nodes includes said processor and at least one each of said memory device and 
said I/O device, and the whole of a plurality of said nodes include at least one 
said processor, at least one said memory device and at least one said I/O device; 
wherein said node controller of each of a plurality of said nodes includes a 
processor unit constituting an interface with said processor in the local node, a 
memory unit constituting an interface with said memory device in the local node 
and/or an I/O unit constituting an interface with said I/O device in the local 
node, a network unit constituting an interface with said inter-node connection 
network, and an intra-node connection circuit for connecting said processor unit, 
at least one of said memory device and said I/O device, and said network unit; 
wherein said processor unit includes an inter-unit address decode circuit and a 
cache coherence control circuit, said inter-unit address decode circuit decoding 
the control information and the address information in said access request, and 
operating in such a manner that the information on the node associated with the 
memory unit or the I/O unit intended as an access request destination, the 
information on the unit intended as an access request destination, and the cache 
coherence control information indicating whether the cache coherence control is 
required or not, are added to a memory access request or an I/O access request 
issued by the processor in the local node, said cache coherence control unit 
performing the cache coherence control of the processor in the local node in the 
case where the cache coherence control information added to the access request 
received from said network unit indicates that the cache coherence control is 
required, ; wherein said I/O unit includes an inter-unit address decode circuit for 
adding the node information and the unit information for the memory unit or the I/O 
unit intended as an access request destination and the cache coherence control 
information indicating whether the cache coherence control is required or not, to 
the memory access request or the I/O access request issued by the I/O device in the 
local node; wherein said network unit includes a transfer unit for transferring the 
access request received from said intra-node connection circuit to said inter-node 
connection network, and a transfer unit for transferring the access request 
transferred thereto from said inter-node connection network to said intra-node 
connection network; wherein said intra-node connection circuit operates in such a 
manner that in the case where a unit in the local node is designated as the 
destination of transfer based on the cache coherence control information, the node 
information and the unit information added to the access request transferred from 
said inter-node connection network through said network unit, said access request 
is transferred to the unit designated as the transfer destination; wherein said 
intra-node connection circuit includes a unit designation circuit for designating 
the destination of transfer of said access request based on the cache coherence 
control information, the node information and the unit information added to the 
access request transferred from said inter-node connection network, and a transfer 
unit for transferring said access request to the unit connected to said intra-node 
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connection circuit in response to the designation of said unit designation circuit; 
wherein said inter-node connection network includes a node designation circuit for 
designating the destination of transfer of said access request based on the cache 
coherence control information, the node information and the unit information added 
to the access request sent out from said network unit, and a transfer unit for 
transferring said access request in response to the designation of said node 
designation circuit; and wherein that access request issued by said processor or 
said I/O device which requires the cache coherence control is broadcast to all the 
nodes requiring the cache coherence control, while the access request not requiring 
the cache coherence control is transferred only to the node intended as a 
destination. 

9. A node controller included in each node of a shared memory multiprocessor 
comprising a plurality of nodes each configured with at least one of a processor 
having a cache memory, a memory device and an I/O device, and a node controller, 
and an inter-node connection network for interconnecting a plurality of said nodes; 
wherein at least one of a plurality of said nodes includes said processor and at 
least one each of said memory device and said I/O device, and the whole of a 
plurality of said nodes include at least one said processor, at least one said 
memory device and at least one said I/O device; wherein said node controller of 
each of a plurality of said nodes includes a processor unit constituting an 
interface with said processor in the local node, at least one of a memory unit 
constituting an interface with said memory device in the local node and an I/O unit 
constituting an interface with said I/O device in the local node, a network unit 
constituting an interface with said inter-node connection network, and an intra- 
node connection circuit for connecting said processor unit, at least one of said 
memory device and said I/O device, and said network unit to each other; wherein 
said processor unit includes an inter-unit address decode circuit and a cache 
coherence control circuit, said inter-unit address decode circuit decoding the 
control information and the address information in said access request, and 
operating in such a manner that the information on the node associated with the 
memory unit or the I/O unit intended as an access request destination, the 
information on the unit intended as an access request destination, and the cache 
coherence control information indicating whether the cache coherence control is 
required or not, are added to a memory access request or an I/O access request 
issued by the processor in the local node said cache coherence control circuit 
performing the cache coherence control of the processor in the local node in the 
case where said cache coherence control information added to the access request 
received from said network unit indicates that the cache coherence control is 
required; wherein said I/O unit includes an inter-unit address decode circuit for 
adding the node information and the unit information for the memory unit or the I/O 
unit intended as an access request destination and the cache coherence control 
information indicating whether the cache coherence control is required or not, to 
the memory access request or the I/O access request issued by the I/O device in the 
local node; wherein said network unit includes a transfer unit for transferring the 
access request received from said intra-node connection circuit to said inter-node 
connection network, and a transfer unit for transferring the access request 
transferred thereto from said inter-node connection network to said intra-node 
connection network; wherein said intra-node connection circuit includes a route 
designation circuit for designating the destination of transfer of the access 
request transferred from said inter-node connection network, based on the cache 
coherence control information, the node information and the unit information added 
to said access request, and a transfer unit for transferring said access request to 
a unit connected to said intra-node connection circuit in response to the 
designation of said route designation circuit; and wherein that access request 
issued by said processor or said I/O device which requires the cache coherence 
control is broadcast to all the nodes requiring the cache coherence control, and 
the access request not requiring the cache coherence control is transferred only to 
the node intended as a destination. 
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13. A shared memory multiprocessor comprising: a plurality of nodes each configured 
with at least one of a processor having a cache memory, a memory device and an I/O 
device; and an inter-node connection network for interconnecting a plurality of 
said nodes; wherein at least one of a plurality of said nodes includes said 
processor and at least one each of said memory device and said I/O device, and the 
whole of a plurality of said nodes include at least one said processor, at least 
one said memory device and at least one said I/O device; wherein each of a 
plurality of said nodes includes an information adding unit for adding the cache 
coherence control information indicating whether the cache coherence control is 
required or not and the node information indicating the node constituting a 
destination of transfer (destination of access), to the access request issued by 
said processor or said I/O device in the local node, and outputting the resulting 
information, and a transfer unit for selectively transferring the access request 
from said information adding unit to said inter-node connection network; and 
wherein said inter-node connection network includes a transfer unit which, based on 
said cache coherence control information and said node information added to the 
access request, transfers said access request to all the nodes requiring the cache 
coherence control among a plurality of said nodes in the case where said cache 
coherence control information indicates that the cache coherence control is 
required, and transfers said access request only to the node indicated by the node 
information added to said access request in the case where said cache coherence 
control information indicates that the cache coherence control is not required. 
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ART-UNIT: 2181 

PRIMARY^EXAMINER: Ray; Gopal C. 
ABSTRACT : 

Each node includes a node controller for decoding the control information and the 
address information for the access request issued by a processor or an I/O device, 
generating, based on the result of decoding, the cache coherence control 
information indicating whether the cache coherence control is 'required or not, the 
node information and the unit information for the transfer destination, ■ and adding 
these information to the access request. An intra-node connection circuit for 
connecting the units in the node controller holds the cache coherence control 
information, the node information and the unit information added to the access 
request. When the cache coherence control information indicates that the cache 
coherence control is not required and the node information indicates the local 
node, then the intra-node connection circuit transfers the access request not to 
the inter-node connection circuit interconnecting the nodes but directly to the 
unit designated by the unit information. 

19 Claims, 16 Drawing figures 
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TITLE: Method and system for maintaining cache coherence in a multiprocessor- 
multicache environment having unordered communication 

Detailed Description Text (4) : 

With reference now to the figures and in particular with reference to FIG. 1, there 
is shown a pictorial representation of a parallel processor or multiprocessor 
computer system which may be utilized to implement the method and system of the 
present invention. Such a computer system would contain at least one shared memory 
10, at least two caching agents capable of maintaining cached copies from the 
shared memory, and an interconnect structure 20 between the memory and the cached 
agents that provides concurrent and unordered transport of communications. The 
caching agent maintaining the cached copies from the shared memory is typically 
either a processor with integral cache or an input/output (I/O) device with 
integral cache. Here, each processor is shown having its own cache , local memory 
and I/O unit, all connected to a common node. 
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Abstract Text - ABTX (1) : 

A multiprocessor system of a distributed shared memory structure has a hot 
plug function for main memories. Each of nodes of the multiprocessor system 
has a processor, an 10 unit, a main memory, a mover, and a routing control 
unit. If a memory read access request is issued from the processor, the 10 
unit^ or the mover to the main memory of a master node t the routing control 
unit instructs the master node to transfer the memory read access request . If 
a memory write access request is issued from the processor, the lO unit, or the 
mover to the main memory of the master node, the routing control unit instructs 
both the master node and a slave node to transfer the memory write access 
request when in a multicasting mode, and instructs only the master node to 



transfer the memory write access request when not in the multicasting mode. 

Claims Text - CLTX (1) : 

1. A multiprocessor system of a distributed shared memory structure having 
a plurality of nodes each comprising at least one processor and a main memory, 
comprising: means for managing the same memory addresses in two nodes ; means 
for holding information of the two nodes as a master node and a slave node; 
means for transferring a read access request from one of a processor and an I/O 
unit with respect to said main memory to a master node ; a flag indicative of 
whether a write access request from one of the processor and the I/O unit with 
respect to said main memory is to be transferred to the master node or to the 
master node and the slave node ; means for transferring the write access 
request to the master node if said flag indicates that the write access request 



is to be transferred to the master node ; means for transferring the write 

access request to the master node and the slave node if said flag indicates 



that the write access request is to be transferred to the master node and the 
slave node ; means for holding address information of a memory space of the 
main memory of each of the nodes ; lock read access means for reading data from 
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ABTX (1) : 

Local memory on a node in a multinode^ multiprocessor computer system with 
distributed shared memory and a remote cache is efficiently updated through the 
use of a combined tag stored in a tag cache. In response to a local processor 
request for access to local memory that does not contain a current copy of the 
data requested, a combined tag is formed from a memory tag and a remote cache 
tag. The combined tag allovws the node to operate in accordance with the 
network protocol such as the Scalable Coherent Interface (SCI) while the memory 
is being updated, acting as memory in response to requests from other nodes to 
the memory and as a cache in response to requests from other nodes to the 
remote cache. In this way the memory is updated quickly and the remote cache 
is not required to store data that is better stored in the local memory. 
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